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ABSTRACT 



A method and apparatus for two-pass image transfor- 
mation, providing a general solution to execute arbi- 
trary warping of an image. A bicubic mesh is created, 
by splines or other suitable means, and is used to create 
displacement tables for X and Y displacement. Alterna- 
tively, the displacement tables can be generated di- 
rectly. The displacement, tables represent the movement 
of each pixel from an original location in the source 
image to a new location in the destination image. One of 
the displacement maps is applied to the source image to 
create an intermediate image and to the other displace- 
ment map to create a resampled displacement map. The 
resampled map is then applied to the intermediate image 
to create the destination image. By resampling, compen- 
sation for altered location points is done automatically. 
In this manner, no inversion of the underlying equations 
and functions is required. 

5 Claims, 5 Drawing Sheets 
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FIG. 2 
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FIG. 3 
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METHOD AND APPARATUS FOR NON-AFFTNE 
IMAGE WARPING 

BACKGROUND OF THE PRESENT INVENTION 5 

1. FIELD OF THE INVENTION 

This invention relates to the field of image processing 
and image transformation. 

2. BACKGROUND ART 

In computer graphics applications, source images are 10 
created and manipulated by a user to achieve a variety 
of effects. A user may. among other operations, rotate, 
invert, animate, distort, resize, color or combine images. 
A computer graphics or imaging system provides a tool 
to accomplish these operations. One method of distort- 15 
ing an image is known as "warping.* 1 

The term warping is often used in image processing 
and computer graphics to mean a two-dimensional 
"resampling" of an image. In general, warping is a map- 
ping of a two-dimensional planar region onto a two-di- 20 
mensional surface. In certain computer graphic applica- 
tions, an image may be mapped onto a two-dimensional 
projection of a three-dimensional surface or portion of a 
surface. 

In many applications, distortions, abberations, or 25 
other flaws are introduced by sensors that are used to 
obtain data to generate a source image. These sensors 
can be any means for capturing or creating a source 
image, such as, for example, lenses, magnetic resonant 
imaging, cat scans, and digitized scans of existing irri- 30 
ages. It is highly desirable to be able to remove imper- 
fections introduced by the sensors from the source im- 
age. Filtering is not always able to remove certain types 
of flaws. Thus, image warping is used. 

In other instances, it may be desired to introduce 35 
distortions to an image for artistic or other reasons. In 
addition, by creating sequences of deformed images, 
each one slighly different than the one before, animated 
sequences can be produced by using warping. 

An image warp, whether to remove or introduce 40 
distortion to a source image, is created and applied to 
the source image. The image warp may be a purely 
mathematical expression of varying degrees of com- 
plexity; (sphere, ellipsoid, etc). The image warp may 
also be a three-dimensional texture representation. 45 

In image processing, warping functions are typically 
characterized or referred to by order of the warping 
function, i.e., an "nth order warp." For example, a "first 
order warp" is a linear function, an example of which is 
a "resize." A second order warp is a quadratic function, 50 
such as a "pin cushion" effect or "barrel" effect. A third 
order warp is described by a cubic polynomial function. 

Warping functions can be wholly arbitrary functions. 
Thus, warping functions are not limited to closed-form 
mathematical descriptions (such as a hemisphere or 55 
other simple form), or to symmetrical forms. For exam- 
ple, a warping function can be described by a flexible 
surface which is pressed or stretched at many locations 
to form an arbitrary three-dimensional surface. This 
arbitrary surface is then applied to an existing image 60 
which is distorted to conform to the irregularities in a 
two dimensional projection of the arbitrary surface. 

PRIOR ART WARPING TECHNIQUES 

One prior art method of executing a warping opera- 65 
tion is known as a "two pass transform" and is described 
in Catmull and Smith's "3-D Transformation of Images 
in Scanline Order," Computer Graphics (Siggraph '80 



proceedings) Vol. 14, No. 3, pp. 279-285, July 1980, and 
incorporated herein by reference. The two pass trans- 
form of this reference spatially transforms 2-D images 
by decomposing the mapping of the image into a se- 
quence of orthogonal, one-dimensional transformations. 

Limitations in this transform technique may be illus- 
trated by a very simple transform, the "affine" trans- 
form. Affine transforms are limited in their ability to 
define complex warps. An affine transform is defined by 
the following matrix: 



[5] 



The matrix specifies the following transformations: 

Y=DX+EY+F 

X and Y represent locations on orthogonal axes. X' 
and Y' are the post transformation locations. A and E 
are scaling factors for X and Y respectively. B and D 
define shears. C and F are used to specify a constant 
offset. 

FIGS. 5A-5D are an example of a very simple trans- 
form. It is a doubling in X and Y. This can be seen in the 
fact that A = 2 and E=2 (FIG. 5D). There is no cross- 
dependence, i.e., X' is independent of Y (B=0), and Y' 
is is independent of X (D=0). 

For A greater than 1 the image is stretched in the X 
direction For Y greater than 1, the image is stretched in 
the Y direction. If A has a value between 0 and 1, the 
image shrinks in the X direction. If E has a value be- 
tween 0 and 1, the image shrinks in the Y direction. By 
making the values negative, the image can be flipped. 

In the first pass, the SOURCE IMAGE (FIG. 5A) is 
transformed to produce TEMPORARY (FIG. SB). 
The displacements are readily observable. Calculation 
shows that the X coordinates in TEMPORARY are 
twice the magnitude of the X coordinates in the 
SOURCE. IMAGE. The Y coordinates are as yet un- 
changed. However, after the second pass when TEM- 
PORARY is transformed, then -the doubling of both X 
and Y coordinates between SOURCE IMAGE and 
DESTINATION can be observed (FIG. 5C). 

Affine transforms can be used for simple geometric 
transformations but do not provide a general solution to 
arbitrary mapping functions. Consider an affine trans- 
form in which there is a Y' dependency on the X term 
(i.e. D^=0), which is often the case. In that situation, 
without transforming Y during the X pass, the second 
pass (Y) equation will yield the wrong effect. During 
the first pass of a two pass transformation, new X values 
are generated. When the second pass is performed, the 
Y' equation requires the original X values. But, only X' 
values are then available. 

A prior art solution to this problem is to invert the 
equation and rcderive the original X values. This can 
easily be accomplished for affine equations. However, 
for higher order warps, the equations are often uncondi- 
tioned, barring easy inversion. In addition, with certain 
warps, it is possible to have a "fold" in the warps so 
certain X values are not retrievable. Therefore, the 
two-pass transform technique utilizing inversion is inad- 
equate for arbitrary warps. 
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Another method used to avoid the step of inverting 
the equation is to maintain the original X locations after 
the first pass and use them as a lookup table to provide 
the appropriate X value for each Y location in the sec- 
ond pass. However, this approach still suffers from a 5 
requirement that the equation must be evaluated in 
forward direction. That is, the warping function must 
be known while the warping operation is performed. 
This does not lend itself to a general warping solution 
and is not easily implemented in hardware, 10 

The prior art methods for performing warping opera- 
tions do not provide, a general solution for arbitrary 
warps. Another disadvantage of prior art warping 
methods is the inability to provide solutions to warps 
having folds. Further, prior art warping methods can 15 
suffer from artifacts, blurring, bottlenecks and other 
problems. 

Therefore, it is an object of the present invention to 
provide a method and apparatus for image transforma- 
tion to implement nth order warps and further arbitrary 20 
warps. 

It is another object of the present invention to pro- 
vide a method and apparatus for image transformation 
which can implement a two-pass algorithm without 
inverting the transfer equations. 25 

Another object of the present invention is to provide 
a method and apparatus for image transformation which 
may be used in connection with warps having folds. 

It is yet another object of the present invention to 
provide a method and apparatus for image transforms- 30 
tion which reduces bottlenecking, blurring and aliasing. 

SUMMARY OF THE PRESENT INVENTION 

The present invention provides a general solution for 
image transformation, such as warping, to realize arbi- 35 
trary mapping functions. The invention also allows a 
warping function to be generated with a bicubic mesh 
defined by splines or some other suitable means. The 
bicubic mesh is utilized to create displacement tables for 
X and Y displacement, respectively. Alternatively, the 40 
tables can be generated directly from a desired warping 
function. The displacement tables represent the move- 
ment of each pixel from an original location in the 
source image to a new location in the destination image. 

One of the displacement tables is applied to the 45 
source image to create an intermediate image and to the 
other displacement table to create a resampled displace- 
ment map. The resampled map is then applied to the 
intermediate image to create the destination image. By 
resampling, compensation for altered source image 50 
points is made. In this manner, no inversion of the un- 
derlying equations is required. 

The present invention also avoids artifacts, blurring, 
bottlenecks and other problems caused by prior art 
warping methods. By considering the area of the inter- 55 
mediate image between the first pass and the second 
pass, the present invention is able to minimize blurring 
and aliasing associated with bottlenecking. Information 
loss is minimized so as to preserve image integrity. This 
intermediate area criteria is applied on a pixel by pixel 60 
basis to ensure that information loss is held at an abso- 
lute minimum. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIGS. 1A and IB illustrate examples of bicubic 65 
meshes. 

FIG. 2 is a flow diagram illustrating the operation of 
the present invention. 



FIG. 3 is a block diagram of the preferred embodi- 
ment of the present invention. 

FIG. 4 is a vector diagram illustrating a local affine 
transformation. 

FIGS. 5A-5D are an example of a two-pass trans- 
form of a simple affine transform. 

DETAILED DESCRIPTION OF THE PRESENT 
INVENTION 

A method and apparatus for image transformation is 
described. In the following description, numerous spe- 
cific details such as number of splines, etc., are set forth 
to provide a more thorough description of the present 
invention. It will be apparent, however, to one skilled in 
the art, that the present invention may be practiced 
without these specific details, In other instances, well- 
known details have not been described in detail so as not 
to obscure the present invention. 

The present invention provides a general solution to 
the transformation of images using arbitrary mapping 
functions. The general solution can be easily imple- 
mented in hardware. 

The present invention utilizes "displacement tables" 
to implement a two pass algorithm which is a general 
solution for arbitrary warps. 

The warping function, however defined, is used to 
identify pixel destination locations in two dimensional 
coordinates. Two displacement tables (also referred to 
as displacement maps), are generated. An X displace- 
ment table is generated which contains the relative 
displacement in the X direction for each pixel in the 
source image when acted on by the warping function. 
The Y displacement table contains the relative displace- 
ment in the Y direction for each source pixel when 
acted on by the warping function. The displacement 
tables are used to provide a numerical solution which 
avoids the problems of prior art warping solutions. 

FIG. 2 is a block diagram of the method of the pres- 
ent invention. Source image 12 is transformed by the 
use of X displacement table 13. This transformation 
creates an intermediate image called TEMP 14, or tem- 
porary image. Simulatneously, Y displacement table 15 
is similarly transformed by X displacement table 13. 
This creates a new version of Y displacement table, 
called Y' displacement table 16. Then, TEMP 14 is 
displaced by Y' displacement table 16. This second 
resampling of the image generates destination image 17, 

Displacement tables 12, 14 and 15 contain the values 
of relative displacement of the pixels in the source im- 
age. The pixel in the "I" column and "J" row will be 
displaced in the X coordinate direction by the value 
contained in the I column and J row of the X displace- 
ment table array. Similarly, that same pixel will be dis- 
placed in the Y coordinate direction by the value in the 
I column and J row of the Y displacement table. 

By applying the X displacement table to the Y dis- 
placement table to create an intermediate Y' displace- 
ment table, the present invention avoids the problem of 
mathematical inversion of complex or ill-conditioned 
functions. 

A displacement table is applied to an image by the use 
of hardware resamplers. The resampler receives as 
input a single column or a single row of the source 
image and the appropriate displacement table. The 
resampler is able to transform the source image one row 
or one column at a time. The process can be instanta- 
neous if a resampler is used for each row (or each col- 
umn) of the source image. The resamplers can be used 



02/18/2004, EAST Version: 1.4.1 



5,175,808 



' in order to transform a displacement table as well. In- 
stead of inputting a row or a column of the source image 
that is to be transformed, a row or column of a displace- 
ment table can be input in its place. The transformation 
will then be effected on the displacement table. 5 

The preferred embodiment of the present invention is 
illustrated in FIG. 3. Source image 12 is coupled on line 
18 to the set of resamplers 21. X displacement table 13 
is coupled on line 19 to sets 21 and 22 of resamplers. Y 
displacement table 15 is coupled on line 20 to set 22 of 10 
resamplers. 

The output 24 of resamplers 21 is provided to TEMP 
image 14. Output 25 of resamplers 22 is provided to Y' 
map 16; Outputs 26 and 27 of TEMP 14 and Y' map 16 
respectively, are provided to resamplers 23. Output 28 15 
of resamplers 23 is provided to destination 17, 

In operation, X displacement table 13 is combined 
with source image 12 at resamplers 21 and with Y dis- 
placement table 15 at resamplers 22. By combining X 
displacement table with Y displacement table 15 at the 20 
first pass, inversion of the underlying warping function 
is not required. Output 24 of resamplers 21 represents . 
the combination of source image 12 and X displacement 
table 13, and is stored in TEMP 14. Output 25 of resam- 
plers 22 represents the combination of X displacement 25 
table 13 and Y displacement table 15 and is stored in Y' 
map 16, representing an intermediate Y value. TEMP 
14 and Y' map 16 are combined in resamplers 23 to 
provide the destination output of the warped transfor- 
mation. 30 

FIG. 3 illustrates the preferred hardware embodi- 
ment when the X pass is done first, with no transposi- 
tion of the X or Y tables, However, with suitable use of 
row and column access a number of configurations of 
FIG. 3 running in parallel can be used to allow for the 35 
different orders of execution required to avoid aliasing 
and bottlenecking problems. 

The source and destination images are composed of 
pixels. During processing, each pixel in the source 
image is treated in one of a number of ways, or a combi- 40 
nation, to create the destination image. A pixel may be 
displaced, i.e., moved to the left, right, up or down. A 
pixel may be compressed. It may be combined with 
neighboring pixels in the source image and mapped to a 
single pixel in the destination image. A pixel in the 45 
source image may be enlarged; i.e. it may be mapped to 
a number of pixels in the destination image. Finally, the 
pixel may disappear during the processing. It may end 
up hidden behind a "fold" in the mapping, or it may. be 
so compressed with so many neighboring pixels, that its 50 
contribution to the image of the destination is negligi- 
ble. 

The processing step between the source and destina- 
tion images is a matter of converting the desired manip- 
ulation, the mapping of the image in terms of numerical 55 
displacements, proportional expansions and compres- 
sions. 

When the requisite displacement tables are generated, 
then the source image is processed in a row-by-row 
manner using the X-coordinate displacement map and 60 
generates an intermediate image called "TEMP". Si- 
multaneously, the Y displacement map is similarly pro- 
cessed to create what is then called Y' displacement 
map. After all the rows of the source image and the Y 
displacement map are thus processed, the intermediate 65 
image "TEMP" is then transformed by the Y' displace- 
ment map to produce the final "DEST" (destination) 
result image of the transformation. 



The displacement mapping can be considered to be of 
two parts, X displacement and Y displacement maps. 
The displacement map can be divided into the X and Y 
composite parts. The X mapping includes an X displace- 
ment for every pixel in the source image. The Y map- 
ping includes a Y displacement for those same pixels. By 
dividing the displacement mapping into X and Y coor- 
dinates, the image manipulator is able to utilize a two- 
pass technique which is a form of image transformation 
that is quick and accurate. The information in both X 
and Y displacement maps is high resolution. It is stored 
in the form of sub-pixel positioning numbers. 

The displacement table method of the present inven- 
tion allows rapid transformation because it avoids the 
time consuming recalculation of source X values. The 
source image is transformed by the X displacement 
map. By applying the X displacement table map to the 
source image and the Y coordinate displacement map, 
the inversion step of prior art methods is eliminated. 
The temporary image, which is the result of an X dis- 
placement map transformation of the source image, is 
then transformed with the Y displacement map which 
has also been transformed by the X displacement map 
(now called Y'). This method numerically adjusts the Y 
displacement map and makes a numerical approxima- 
tion of the original X values unnecessary. It is much 
quicker than doing the calculations required to invert 
the original mapping. Furthermore, with some ill-con- 
ditioned displacement maps, the inversion is sometimes 
impossible. Therefore, the numerical approach embod- 
ied by this two-pass transformation technique solves 
problems which are otherwise insurmountable when 
doing inversion-based transformation. 

The present technique also includes appropriate in- 
terpolation and accumulation functions. When a num- 
ber of pixels are to be mapped onto a larger number of 
pixels of the destination image, interpolation takes place 
to cover the greater area. The interpolation function is 
able to cover the undefined pixels in the destination 
image by a smooth transition between defined pixels. 
Conversely, the accumulation function is able to map a 
number of pixels in the source image to a single pixel in 
the destination image. The accumulation function accu- 
mulates the values of the plurality of pixels to be 
mapped to the single destination pixel and sums the 
weighted contributions to create the value of the desti- 
nation pixel. 

WARP FUNCTION 

In one embodiment of the present invention, a bicubic 
mesh is utilized to create the image warp. The bicubic 
mesh is generated by a matrix of knots, which defines a 
mesh of splines. A spline is defined by four or more 
points in space. The points, referred to as "knots," are 
weighted dependent on a basis function which, controls 
the interpolation between knots. A four-by-four array 
of knots defines a two-dimensional spline and more 
knots can be used to define more complex splines. 

Referring to FIG. 1A, a bicubic mesh of unity is 
Ulustrated. The mesh 10 is comprised of a plurality of 
knots HA- 11 P. In a computer graphics environment, 
the mesh of FIG. 1A is displayed over a source image. 
Because the image of FIG. 1A is a mesh of unity, no 
distortion of the underlying source image takes place. 

Using a cursor or other screen input device, a user 
can manipulate the knots, deforming the mesh as de- 
sired. For example, the mesh of FIG. IB was created by 
moving knots 11F, 11J, UK, and IIP of FIG. 1A. The 
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deformed bicubic mesh is then used to create displace- 
ment maps for altering the source image to conform to 
the bicubic mesh. 

The above method is very useful because of its ability 
to create a surface based on a limited number of surface 
changes. A user may move any number of knots in a 
manner which reflects his desire to distort the source 
image. A user may add more knots as required for 
greater control. A surface is then generated which 
makes the distortion continuous by interpolating be- 
tween the desired distortion points. The interpolation 
function is independent of the warping function. In the 
preferred embodiment of the present invention, the 
spline is a Catmull-Rom spline as defined in Catmull and 
Rom, **A Class of Local Interpolating Splines," Com- 
puter Aided Geometric Design, R. E. Bamhill and R. F. 
Riesenfeld, ed. Academic, New York 1974. 

Using this method, a user need not define a complex 
mathematical function to attempt to distort his image. 
The user need only move a number of knots on a map- 
ping from which the computer can derive a spline sur- 
face and then calculate the necessary displacement, 
compression, expansion or erasure of the composite 
pixels of the source image. A user may utilize a mapping 
with many knots in order to precisely define image 
distortion. A user may also define a mapping with very 
few knots in order to introduce distortion to an image 
which is less geographically precise. 

Once a mathematical function has been defined or 
knots have been moved and an approximating spline 
generated, then a displacement map is derived from the 
mathematical mapping, or model. The displacement 
map generated describes an X direction displacement 
and a Y direction displacement for each pixel in the 
source image. The displacement map contains displace- 
ment information to sub-pixel accuracy. 

OPTIMIZATION 

A problem that can occur in two-pass transforma- 
tions is loss of data resolution. If the first pass com- 
presses the image and the second pass enlarges it, pixel 
resolution is compromised. The present invention 
avoids this problem by making a pixel by pixel decision 
on how to preserve resolution by executing the enlarg- 
ing step first, if possible. The present invention con- 
siders the area of the intermediate image between the 
first pass and the second pass. 

The transformation can be done by applying the X 
displacement table first. Or, it may be achieved by per- 
forming the Y displacement table pass first. The source 
image and the mapping tables may be first transposed in 
order to reduce aliasing and blurring. And after the 
source image is transposed, then the X pass, or the Y 
pass can be executed first. In the present invention, the 
determination of whether an image should be X-passed 
first or Y-passed first, transposed or untransposed 
source image, can be done on a pixel by pixel basis. This 
ensures that every pixel is transformed in the most effec- 
tive manner and the least amount of information is lost 

In combination, there are four different sequences in 
which the two-pass technique can be applied. Either the 
X or Y displacement table can be applied first and the 
source image and X and Y displacement tables can be 
applied in original or transposed form. Therefore, the 
four different approaches to applying the two-pass tech- 
nique are: (1) X displacement table application first; (2) 
Y displacement table application first; (3) transpose 
source and displacement tables, X table application first; 
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and (4) transpose source and displacement tables, Y 
table application first. The determination as to. which of 
these four approaches best accomplishes the transfor- 
mation depends on a criteria of intermediate surface 
5 area. The goal is to minimize information loss both due 
to compression in the first pass which must be expanded 
in the second pass and excessive shearing in the first 
pass. 

The criteria for determining which of the four meth- 
10 ods is most useful in transforming the image, that is, 
which of the four approaches minimizes artifacts, is 
based on the criteria of intermediate surface area, the 
intermediate surface area of the temporary image. The 
temporary image is the image which is obtained after 
,5 . the source image has been transformed by a first pass 
(which can be from either the X displacement map or 
the Y displacement map, or transposed versions 
thereof). 

In operation, the displacement maps are analyzed to 

20 determine the intermediate area which would be gener- 
ated for each of the four transformation orders. 

Of the four temporary images which could be gener- 
ated, the one with the greatest surface area represents 

25 the first pass which optimizes information retention. 
That is, if applying the X displacement map to the 
source image creates an intermediate image of surface 
area Si, applying the Y displacement map to the source 
image creates an intermediate image of surface area S2, 

30 applying the transposed X displacement map to the 
transposed source image creates an intermediate image 
of surface area Si, applying the Y displacement map to 
the source image creates an intermediate image of sur- 
face area S2, applying the transposed X displacement 

35 map to the transposed source image creates an interme- 
diate image of surface area S3, applying the transposed 
Y displacement map to the transposed source image 
creates intermediate image with surface area S4. Infor- 
mation loss is minimized by considering the relative 

4Q values of S], S2, S3 and S4. If S2 is greater than Si, S3 and 
S4, then the algorithm of the present invention applies 
the untransposed Y displacement map to the untran- 
sposed source image in order to optimize information 
retention. If S3 is greater than Si, S2 and S4, then the 

45 source image and displacement maps should first be 
transposed, and then the transposed X displacement 
map applied first to the transposed source image. If S] is 
the greatest of the four surface areas, then the untran- 
sposed X displacement map should be first applied to 

50 the untransposed source image. If S4is the greatest, then 
the source image and displacement maps should be first 
transposed, and then the Y displacement map applied to 
the source image. 
So far, this consideration as to X or Y displacement 

55 table application first or transposed or untransposed 
tables and images has limited itself to a criteria based on 
the entire image. Optimization can be at a pixel level as 
well. That is, each pixel can be evaluated by the surface 
area to which that pixel maps in the destination image. 

60 By this technique, each pixel is determined to be opti- 
mally transformable in one of the four above methods. 
Each pixel is evaluated as described in the previous 
paragraph and its optimum method for transformation is 
noted and stored. Ultimately, when the transformation 

65 is executed, the algorithm utilizes the information for 
each pixel and transforms that particular pixel in the 
optimum of the four possible application approaches. 
The calculations for A, B, C and D are performed for 
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each pixel. Then the determination of order of execu- are also 0. Similarly, if a pixel in the source image is to 

tion can be made to eliminate artifacts. be transformed in the transposed form with the Y pass 

The. present invention assumes that the transforma- first, then the corresponding positions in the X pass first, 

tion is locally affine at a pixel. Thus, only three pixels y paS s first and X transpose first will all be 0. Only the 

need to be considered, the current pixel and its immedi- . 5 corresponding pixel in the Y transpose first image will 

ate right and lower neighbors. FIG. 4 illustrates this ^ e \ 

situation. Notice that d' is implicit, since we are assum- T ^ destination image, denoted W, is given by a 
ing a local affine approximation. Since we have a map , inear combination of four images: 
for each pixel, a, b, and c are on the unit lattice. Exami- 
nation of the affine equation tells us that: io iw-Wxrc+fxTtrxFc+WrFv+lrTtTYFc 



A=aV x 
B-aV, 



Ijr is the source image transformed X pass first. I y is 
the source image transformed Y pass first, ljryis the 
source image transformed X first in transposed form. 
C^&'b' y ,15 I j7*is the source image transformed Y first in transposed 

form. The equation above can be read. \ 0 uu the destina- 
D - a c V tion image, is the sum of: the source image transformed 

X pass first multiplied by the X first control image 
Our map stores AX and AY, which gives us the fol- q^ V¥) ^ trans f orme d X pass first; the source image 
lowing: 20 transformed X pass first in the transposed form multi- 

a' =a +Ao by tne x transposed first (XPF) control image also 

° x ° x 0jr transformed with the X pass first in the transposed form; 

v x =b x +Ab x the source image transformed Y pass first multiplied by 

the Y pass control image (XPF) also processed Y pass 
c'x=cx-t*c x 25 first; and the source image transformed Y pass first in 

the transposed form multiplied by the Y transposed 
control image (YTF) also transformed in the Y pass first 
transposed form, lout, the destination image is the sum of 
these four products. By utilizing this number of addi- 
ct <v+ Af ,. tional warps and images, the maximum amount of infor- 
mation is retained. This minimizes undesirable effects 
Knowing that a, b and c are on the unit lattice gives such as blurring, aliasing and the production of artifacts, 
us: If desired, only two control images, X pass first and 

35 transpose, need be generated and Iout is generated as 
t>x=a x +\ follows: 



c.x=a. T 



40 



sw- todi -/rc>+ lxrlxdTC+W -/jtcK- 
i-ird+W-txcVTC 



DISPLACEMENT MASKS 



Thus we have- The P resent invention also provides a displacement 

mask for use in warping operations. With a displace- 
/*=a6 x -Aj.,+ i ment mask an image manipulator may specify how 

45 much of the warping is to be applied at each pixel. It 
5= Ac, - La x may reduce or eliminate warping over a greater portion 

of the image, ensuring that only a very small portion of 
D=Lb y -La y it i$ actua ]i y warped. For example, to transform the 

E—Lc — Afl +1 image of a face, the user can specify that only the mouth 

y ~ y 50 region is to be transformed. The rest of the source 

Information is gathered on a pixel by pixel basis. For image remains unchanged, even if the warp specified is 
each pixel, the present invention determines whether global. A displacement mask may be thought of as a 
the transformation should occur X first, Y first and in scaling of a displacement map. It transitions between 
transposed or untransposed form as follows. First the 53 regions where the specified warping is to be applied 
absolute values of A, B, D and £ are determined! If completely to those regions where little or none of the 
either of A or B is greater than both of D or E» perform effect of the warp is executed, 
the X pass first, otherwise, perform the Y pass first. If SHADING 
the X pass is first, and A is less than B, then also perform artAUiiNU 
a transposition. If Y pass is first, and E is less than D, ■ Because the method of the present invention utilizes 
then also perform a transposition. Information is stored relative displacements, a gradient field exists for the 
in four images. Each image is composed of Ts and 0*s. image. If a z field is present, a surface normal A can be 
For any given pixel, if that pixel is to be transformed X directly computed. Even if only x and y information is 
pass first, then the corresponding position in the X pass present, a surface normal can still be generated. M re- 
first (XPF) image is a 1, and the Y pass first, (YPF) ^ fl ects tne fact that the gradient field is normalized, 
image, is a 0. Also, the corresponding positions in the X M= || du, dv, da || depends on the way in which the 
transpose (XTF) image and Y transpose (YTF) image fie j d ^ normalized. 
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\du+dw\<M 

Thus, dz should be defined such that: 
dtr+dMttr^M 2 

Since, 
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II VC|| 



and 

||VG||=Masdefmed 

and the computation of dz completely specifies VG, 
then we can compute A as: 



10 



15 



(dv. 



Given this unit normal vector to the surface at a pixel, 20 
a shading function can be invoked at each pixel. The 
shading method is described in U.S. Pat. No. 4,835,712 
entitled "METHOD AND APPARATUS FOR IM- 
AGING VOLUME DATA WITH SHADING" and 
assigned to the assignee of the present invention, and 25 
incorporated herein by reference. Other shading func- 
tions can also be easily implemented such as surface 
material properties. 



FOLDS 



30 



A bicubic warp may have "folds" in it, where the 
surface can be thought to fold back on itself. If a 2 map 
is present, then it can be determined which parts of the 
surface are visible. Without a Z buffer, some arbitrary 
assumptions can be made which allow the present in- 35 
vention to produce coherent output. The present inven- 
tion utilizes a moving horizon algorithm. The displace- 
ment table is scanned from the left to right and a hori- 
zon maintained. Pixels are only "visible" if they move 
the horizon. Otherwise, they are lost behind the hori- 40 
zon. This approach assumes that an obscured area stays 
obscured. This provides a reasonable solution to the 
fold problem without requiring large amounts of mem- 
ory. 

Thus, a general solution to implement arbitrary warp- 45 
ing functions has been described. 
1 claim: 

1. A method for performing a non-affine image trans- 
formation on a source image comprised' of a plurality of 
points having at least X and Y values comprising the 50 
steps of: 

defining a warp mesh representing said non-af!ine 
transformation of said source image; 

defining an X displacement table, the entries of said X 
displacement table representing the displacement 55 
of each point of said source image in an X direction 
resulting from said non-affine transformation; 

defining a Y displacement table, the entries of said Y 
displacement table representing the displacement 
of each point of said source image in a Y direction 60 
resulting from said non-affine transformation; 

resampling said source image with said X displace- 
ment table to generate an intermediate image; 

resampling said Y displacement table with said X 
displacement table to generate an intermediate Y 65 
displacement table; 

resampling said intermediate image with said interme- 
diate Y displacement table to generate a destination 



image representing said non-affine transformation 
of said source image. 

2. The method of claim 1 wherein said entries of said 
X and Y displacement tables are described to sub-pixel 
accuracy. 

3. A circuit for performing a non-affine image trans- 
formation on a source image comprised of a plurality of 
points having at least X and Y values comprising: 

first storage means for storing said source image; 

second storage means for storing an X displacement 
table, the entries of said X displacement table rep- 
resenting the displacement of each point of said 
source image in an X direction resulting from said 
non-affine transformation; 

third storage means for storing a Y displacement 
table, the entries of said Y displacement table repre- 
senting the displacement of each point of said 
source image in a Y direction resulting from said 
non-affine transformation; 

first resampling means coupled to said first and sec- 
ond storage means for resampling said source 
image with said X displacement table to generate 
an intermediate image, said intermediate image 
stored in a fourth storage means; 

second resampling means coupled to said second and 
third storage means for resampling said Y displace- 
ment table with said X displacement table to gener- 
ate an intermediate Y displacement table, said inter- 
mediate Y displacement table stored in a fifth stor- 
age means; 

third resampling means coupled to said fourth and 
fifth storage means for resampling said intermedi- 
ate image with said intermediate Y displacement 
table to generate a destination image representing 
said non-affine transformation of said source image, 
said destination image stored in a sixth storage 
means. 

4. A method for performing a non-affine image trans- 
formation on a source image comprised of a plurality of 
points having at least X and Y values comprising the 
steps of: 

defining a warp mesh representing said non-affine 
transformation of said source image; 

defining an X displacement table, the entries of said X 
displacement table representing the displacement 
of each point of said source image in an X direction 
resulting from said non-affine transformation; 

defining a Y displacement table, the entries of said Y 
displacement table representing the displacement 
of each point of said source image in a Y direction 
resulting from said non-affine transformation; 

resampling said source image with said X displace- 
ment table to generate a first intermediate image of 
surface area SI; 

resampling said source image with said Y displace- 
ment table to generate a second intermediate image 
of surface area S2; 

transposing said source image and said X and Y dis- 
placement tables, and resampling said transposed 
source image with said transposed X displacement 
table to generate a third intermediate image of 
surface area S3; 

resampling said transposed source image with said 
transposed Y displacement table to generate a 
fourth intermediate image of surface area S4; 

when Si is greater than $2, S3 and S4, resampling 
said Y displacement table with said X displacement 



02/18/2004, EAST Version: 1.4.1 



table to generate an intermediate Y displacement 
table, then resampling said first intermediate image 
with said intermediate Y displacement table to 
generate a destination image representing said non- 
affine transformation of said source image; 5 

when S2 is greater than SI, S3 and S4, resampling 
said X displacement table with said Y displacement 
table to generate an intermediate X displacement 
table, then resampling said second intermediate 
image with said intermediate X displacement table 10 
to generate a destination image representing said 
non-affme transformation of said source image; 

when S3 is greater than SI, S2 and S4, resampling 
said transposed Y displacement table with said 
transposed X displacement table to generate a IS 
transposed intermediate Y displacement table, then 
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resampling said third intermediate image with said 
transposed intermediate Y displacement table to 
generate a destination image representing said non- 
aftine transformation of said source image; 
when S4 is greater than SI, S2 and S3, resampling 
said transposed X displacement table with said 
transposed Y displacement table to generate a 
transposed intermediate X displacement table, then 
resampling said fourth intermediate image with 
said transposed intermediate X displacement table 
*to generate a destination image representing said 
non-affme transformation of said source image. 
5. The method of claim 4 wherein said entries of said 

X and Y displacement tables are described to sub-pixel 

accuracy. 

***** 



25 



35 



40 



45 



50 



55 



60 



65 



02/18/2004, EAST Version: 1.4.1 



